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Abstract 

We  have  been  studying  DoD  acquisition  decision-making  since  2009.  The  U.S.  DoD 
acquisition  process  is  extremely  complex.  There  are  three  key  processes  that  must  work  in 
concert  to  deliver  capabilities:  determining  warfighters’  requirements/needs,  the  DoD  budget 
planning,  and  the  procurement  of  final  products.  Each  process  produces  large  amounts  of 
information  (Big  Data).  There  is  a  critical  need  for  automation,  validation,  and  discovery  to 
help  acquisition  professionals,  decision-makers,  and  researchers  understand  the  important 
content  within  large  data  sets  and  optimize  DoD  resources  throughout  the  processes.  Lexical 
Link  Analysis  (LLA)  can  help,  by  applying  automation  to  reveal  and  depict — to  decision¬ 
makers — the  correlations,  associations,  and  program  gaps  across  all  or  subsets  of  acquisition 
programs  over  many  years.  This  enables  strategic  understanding  of  data  gaps  and  potential 
trends,  and  can  inform  managers  where  areas  might  have  higher  program  risk  and  how 
resource  and  big  data  management  might  affect  the  desired  return  on  investment  among 
projects.  In  this  paper,  we  describe  new  developments  in  analytics  and  visualization,  how 
LLA  is  adaptive  to  Big  Data  Architecture  and  Analytics  (BDAA),  and  needs  for  Big  Acquisition 
Data  used  in  Defense  Acquisition  Visibility  Environment  (DAVE). 
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Background 

We  have  been  studying  Department  of  Defense  (DoD)  acquisition  decision-making 
since  2009  (Gallup  et  al.,  2009;  Zhao,  Gallup,  &  MacKinnon,  201 0,  201 1 , 2012a,  2012b, 
2013,  2014).  The  U.S.  DoD  acquisition  process  is  extremely  complex.  There  are  three  key 
processes  that  must  work  in  concert  to  deliver  capabilities:  definition  of  warfighters’ 
requirements/needs,  DoD  budget  planning,  and  procurement  of  products,  as  in  Figure  1 . 
Each  process  produces  volumes  of  information  (Big  Data).  The  need  for  automation, 
validation,  and  discovery  is  now  a  critical  need,  as  acquisition  professionals,  decision¬ 
makers,  and  researchers  grapple  to  understand  data  and  make  decisions  to  optimize  DoD 
resources. 


How  to  buy:  bitaxtfif 
cost,  schedule, 
performance 


UA  jutorrutrs  the  posvibiltty  to  develop  awjreneii  of  the 
"fit"  between  the  budget,  riiwl  products,  <irvd  requirementv. 
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Figure  1.  DoD  Acquisition  Decision-Making 

Since  2009,  we  have  been  working  on  the  problem  of  how  the  interlocking  systems 
processes  become  aware  of  their  fit  between  DoD  programs  and  warfighters’  needs.  How 
are  gaps  revealed?  Moreover,  in  the  performance  of  DoD  acquisition  processes,  each 
functional  community  is  required  to  review  only  the  particular  information  for  which  it  is 
responsible,  further  exacerbating  the  problem  of  lack  of  fitness.  For  example,  the  systems 
engineering  community  typically  only  examines  the  engineering  documents  and  feasibility 
studies,  the  test  and  evaluation  community  looks  only  at  the  test  and  evaluation  plans,  and 
the  acquisition  community  looks  at  the  acquisition  strategies.  Rarely  do  these  stakeholders 
review  each  other  data  or  jointly  discuss  the  core  questions  and  integrated  processes 
together  as  shown  in  Figure  1 . 

Motivated  by  this  lack  of  fit  and  horizontal  integration,  we  have  been  applying  Lexical 
Link  Analysis  (LLA),  a  data-driven  automation  technology  and  methodology  across  DoD 
acquisition  processes  to 

•  surface  themes  and  their  relationships  across  multiple  data  sources 

•  discover  high  value  areas  for  investment 

•  compare/correlate  data  from  multiple  data  sources 

•  sort/rank  important  and  interesting  information 

LLA  is  a  data-driven  method  for  pattern  recognition,  anomaly  detection,  and  data 
fusion.  It  shares  indexes  not  data,  feasible  for  parallel  and  distributed  processing,  adaptive 
to  Big  Data  Architecture  and  Analytics  (BDAA)  and  needs  for  Big  Acquisition  Data. 
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As  an  example  from  past  work,  we  took  a  detailed  look  at  the  Research, 
Development,  Test  and  Evaluation  (RDT&E)  budget  modification  practice  from  one  year  to 
the  next  over  the  course  of  10  years  and  about  450  DoD  program  elements.  We  found  a 
pattern  that  the  programs  with  fewer  links  (measured  by  LLA)  to  warfighters’  requirements 
received  more  budget  reduction  in  total  but  less  on  average,  indicating  the  budget  reduction 
may  have  focused  only  on  large  and  expensive  programs  rather  than  perhaps  cutting  all  the 
programs  that  do  not  match  warfighters’  requirements.  Furthermore,  the  programs  with  more 
links  to  each  other  received  more  budget  reduction  in  total,  as  well  as  on  average,  indicating 
a  pattern  of  good  practice  of  allocating  DoD  acquisition  resources  to  avoid  overlapping 
efforts  and  to  fund  new  and  unique  projects.  These  findings  were  useful  as  validation  and 
guidance  for  future  decision  processes  for  automatically  identifying  programs  to  match  the 
warfighter’s  requirements,  limit  overall  spending,  minimize  efficiencies,  eliminate 
unnecessary  cost,  and  maximize  the  return  on  investment. 

In  this  paper,  we  demonstrate  a  set  of  comprehensive  LLA  analysis  reports  and 
visualizations  generated  automatically  from  multiple  data  sources.  These  reports  and 
visualizations  reveal  data  correlations  and  gaps  among  multiple  data  sources.  These 
correlations  and  gaps  could  form  the  basis  for  pattern  recognition,  anomaly  detection,  and 
further  inquiry  or  future  reconciliation  of  the  expectations  (e.g.,  acquisition  strategy)  and 
realities  (e.g.,  engineering  feasibility)  from  various  communities.  The  automatic  discovery  of 
the  disconnection  or  gaps  could  be  fed  back  to  the  human  analysts  or  decision-makers  for 
decision-making  and  resource  management. 

Methodology 

Lexical  Link  Analysis  (LLA) 

LLA  has  been  used  to  analyze  unstructured  and  structured  data  for  pattern 
recognition,  anomaly  detection,  and  data  fusion.  It  uses  the  theory  of  system  self-awareness 
(SSA)  to  identify  high-value  information  in  the  data  that  can  be  used  to  guide  future  decision 
processes  in  a  data-driven  or  unsupervised  learning  fashion.  It  is  implemented  via  a  smart 
infrastructure  named  “system  and  method  for  knowledge  pattern  search  from  networked 
agents  (U.S.  patent  8,903,756),”  also  known  as  Collaborative  Learning  Agents  (CLA), 
licensed  from  Quantum  Intelligence,  Inc.  (Zhou,  Zhao,  &  Kotak,  2009). 

In  LLA,  a  complex  system  is  expressed  in  specific  vocabularies  or  lexicons  to 
characterize  its  features,  attributes,  or  surrounding  environment.  LLA  uses  bi-gram  word 
pairs  as  the  features  to  form  word  networks.  Figure  2  depicts  using  LLA  to  analyze  10  years 
of  reports  in  the  Naval  Postgraduate  School  (NPS)  Acquisition  Research  Program  with  word 
pairs  as  groups  or  themes.  Figure  3  shows  a  detail  of  a  theme  in  Figure  2.  A  node 
represents  a  word.  A  link  or  edge  represents  a  word  pair. 
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Figure  2.  Themes  Discovered  in  Colored  Groups 


Figure  3.  A  Detailed  View  of  a  Theme  in  Figure  2 

LLA  is  related  to  bags-of-words  (BAG)  methods  such  as  LDA  (Blei,  Ng,  &  Jordan, 
2003)  and  text-as-network  (TAN)  methods  such  as  the  Stanford  Lexical  Parser  (SLP; 
Stanford  Natural  Language  Processing  Group  [SNLPG],  2015).  LLA  selects  and  groups 
features  into  three  basic  types: 

•  Popular  (P):  They  are  the  main  themes  in  the  data.  Figure  3  is  an  example  of 
a  popular  theme  centered  around  word  nodes  “analysis,”  “model,”  and 
“approach.”  These  themes  could  be  less  interesting  because  they  are  already 
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in  the  public  consensus  and  awareness.  They  represent  the  patterns  in  the 
data. 

•  Emerging  (E):  Themes  may  grow  to  be  popular  over  time.  Figure  4  is  an 
example  of  an  emerging  theme  centered  around  word  nodes  “national,” 
“defense,”  and  “acquisition.” 

•  Anomalous  (A):  These  themes  may  be  off-topic  themes  that  are  interesting 
for  further  investigation.  Figure  5  is  an  example  of  anomalous  theme  centered 
around  word  nodes  “stock”  and  “market(s).” 


Figure  4.  An  Example  of  Emerging  Theme 


Figure  5.  An  Example  of  Anomalous  Theme 

Figure  6  summarizes  LLA  used  for  historical  and  new  data.  The  red  part  shows  a 
pattern  (e.g.,  a  theme)  discovery  phase  using  historical  data  including  data  fusion  that  come 
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from  multiple  learning  agents.  The  black  part  shows  an  application  phase  that  new  data  is 
compared  with  the  patterns  discovered  and  hence  the  anomalies  are  revealed. 


Figure  6.  Diagram  for  the  LLA  Method 

Word  Pairs  Generalization  and  CCC  Method 

Figure  7  shows  the  word  pairs/bi-gram  in  an  LLA  can  be  generalized  as  a  Context- 
Concept-Cluster  (CCC)  model,  where  a  context  is  a  generic  attribute  that  can  be  shared  by 
multiple  data  sources,  a  concept  is  a  specific  attribute  for  a  data  source,  and  a  cluster  is  a 
combination  of  attributes  or  themes  that  can  be  computed  using  a  word  community  finding 
algorithm  (e.g.,  Girvan  &  Newman,  2002)  in  Figure  6  to  characterize  a  data  set.  Context  can 
be  a  word,  location,  time,  or  object,  and  so  on. 
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Using  clustered  attributes 

•  Describe  themes 

•  Describe  patterns 

•  Discover  anomalies 

•  Retrieve/refer  objects 

•  Classification  and  prediction 


Figure  7.  The  Word  Pairs/Bi-Gram  in  LLA  a  Context-Concept-Cluster  (CCC)  Model 

Figure  8  summarizes  how  a  generalized  CCC  method  is  used  for  historical  and  new 
data.  Similar  to  Figure  6,  there  is  a  pattern  discovery  phase  using  historical  data  where 
patterns  are  learned  and  discovered,  and  an  application  phase  for  a  new  data  is  compared 
with  the  patterns  discovered,  and  anomalies  are  revealed. 


Figure  8.  Diagram  for  the  CCC  Model 
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Research  Results 
Taskl 

We  are  working  with  the  OSD  OUSD(ATL)  (US)  to  install  the  LLA/SSA/CLA  system 
as  a  web  service  using  a  Linux  platform  (i.e.,  CentOS)  in  the  Defense  Acquisition  Visibility 
Environment  (DAVE)  test  bed.  We  created  a  publically  available  data  set  with  the  installation 
to  test.  In  this  example,  data  sources  include  10  days  of  business  news  of  about  1 ,000 
companies,  which  are  organized  in  industries  as  follows: 

•  Technology 

•  Services 

•  Healthcare 

•  Utilities 

•  Basic  Materials 

•  Financial 

•  Consumer  Goods 

•  Industrial  Goods 

•  Conglomerates 

Each  category  of  information  such  as  “Healthcare”  or  “Consumer  Goods”  are 
indexed,  mined,  and  listed  under  “Index  Management”  separately  in  a  single  LLA  server. 
When  clicking  “Fuse,”  these  indexed/mined  models  are  fused  into  one  model.  Figure  9 
shows  “Fuse  Results”  from  LLA  listed. 


Figure  9.  Fuse  Results  Listed 

Figure  10  shows  the  discovered  themes,  where  green  themes  101  (P)  and  20(P)  are 
“popular”  themes,  blue  themes  156(E),  49(E),  and  46(E)  are  “emerging”  themes,  and  gold 
themes  208(A),  62(A),  and  others  are  “anomalous”  themes. 

•  Popular  themes  are  the  main  themes  in  the  data.  Figure  11  is  an  example  of 
a  popular  theme  centered  “dividend  cuts,  see  dividend”  for  this  data.  Columns 
“Consensus”  is  the  ratio  of  the  number  of  matched  word  pairs  (i.e.,  at  least 
two  data  sources  contain  the  word  pairs)  over  the  number  of  unique  word 
pairs  (i.e.,  only  one  data  source  contains  the  word  pairs).  These  themes 
could  be  less  interesting  because  they  are  already  in  the  public  consensus 
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and  awareness.  They  represent  the  patterns  in  the  data.  The  red  links 
represent  the  word  pairs  that  are  shared  for  at  least  two  data  sources  while 
the  black  data  sources  are  unique  to  one  data  source. 

•  Emerging  themes  may  grow  to  be  popular  over  time.  Figure  12  is  an  example 
of  an  emerging  theme  centered  “back  shares,  Canada  back.” 

•  Anomalous  themes  may  be  off-topic  themes  that  are  interesting  for  further 
investigation.  Figure  13  is  an  example  of  anomalous  theme  centered  around 
“top  buys,  set  top.”  Anomalous  concepts  are  more  interesting  to  investigate, 
for  example,  concepts  in  Figure  13  such  as  “buys  web,”  “streaming  service,” 
“buys  insider,”  “web  ipo,”  and  so  on,  may  have  better  returns  on  investment 
than  the  concepts  in  a  popular  theme  such  as  “sees  dividend”  and 
“announces  positive.” 
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Figure  10.  Discovered  Themes  Listed 
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Figure  12.  Emerging  Themes  (e.g.,  156[E]) 
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Figure  13.  Anomalous  Theme  (e.g.,  208[A]) 

Match  Matrix  Unique  Word  Pairs  by  Theme 

Figure  14  shows  the  numbers  of  unique  word  pairs  in  a  data  source  and  a  theme. 
For  example,  there  are  12  unique  word  pairs  for  the  data  source  “lndex_BasicMaterials”  in 
the  theme  titled  “101:dividend  cuts,  sees  dividend”  in  Figure  14.  Clicking  this  number  leads 
to  a  list  showing  the  12  word  pairs  as  shown  in  Figure  15.  Figure  16  shows  the  list  can  be 
further  drilled  down  to  a  search  result  list  (e.g.,  “sees  energy”)  and  to  the  original  documents 
that  contain  the  word  pair. 
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Figure  14.  Match  Matrix  Unique  Word  Pairs  by  Theme 
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Figure  15.  Drill-Down  List  for  the  Unique  Word  Pairs  in  Theme  101  and 

Index  BasicMaterial 
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Figure  16.  Drill-Down  Search  on  “Sees  Energy” 
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Match  Matrix 

Figure  17  shows  the  match  matrix  for  comparing  data  sources.  The  column  “Match 
Score”  shows  the  number  of  matched  word  pairs  for  lndex_BasicMaterials.  “5.00(0.02)” 
shows  the  number  (5)  of  matched  word  pairs  and  correlation  (0.02)  between 
lndex_BasicMaterials  and  lndex_Financial.  The  correlation,  computed  as 
=5/(sqrt(30+223)*sqrt(28+270)),  is  normalized  using  the  match  scores  and  uniqueness 
scores  for  both  data  sources.  Clicking  on  the  “5.00(0.02)”  leads  to  the  list  of  the  matched 
word  pairs  for  the  two  sources  as  shown  in  Figure  18.  Clicking  on  “Energy  Prices”  or  “Oil 
Fund”  (i.e.,  the  red  boxes  in  Figure  18)  leads  to  the  search  results  of  two  terms  respectively. 
The  search  results  are  sorted  in  a  descending  order  of  the  counts  of  how  many  “popular,” 
“emerging,”  and  “anomalous”  word  pairs  appear  in  the  original  documents.  For  example, 
some  marketing  applications  may  need  listing  the  popular  terms,  and  business  intelligence 
applications  may  need  listing  anomalous  terms  as  shown  in  Figure  19  (a)  and  (b) 
respectively.  Clicking  the  “vis”  link  in  Figure  19  (a)  and  (b)  lead  to  the  corresponding  themes 
to  which  these  word  pairs  (e.g.,  “energy  prices”  and  “oil  fund”)  belong. 


Figure  17.  Match  Matrix  and  Visualization  List 


Figure  18.  Drill-Down  (e.g.,  Correlation  Between  lndex_BasicMaterials  and 

!ndex_Financial) 
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Figure  19.  Drill-Down  Options  From  Figure  10 

Figure  17  also  includes  a  list  of  D3  visualizations  implemented.  Figure  20  shows  a 
D3  network  visualization  for  all  the  data  sources;  their  connections  among  the  nodes  are 
computed  based  on  the  correlations  from  the  lexical  links  in  Figure  17.  The  node 
connections  represent  all  the  correlations:  thicker  (thinner)  connections  indicate  higher 
(lower)  correlations.  The  clusters  are  generated  based  on  the  correlations.  Figure  21  shows 
a  D3  correlation  matrix  view  of  all  the  data  sources.  Figure  22  shows  a  D3  time-series 
bubble  chart,  which  depicts  the  changing  of  the  themes  over  time. 
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D3  Time  Series  Bubble  Chart 
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Figure  22.  D3  Time-Series  Bubble  Chart 

Task  2 

We  are  also  exploring  how  to  use  LLA  jointly  with  other  business  intelligence  tools, 
especially  Big  Data  Architecture  and  Analytics  (BDAA)  tools: 

•  Deep  learning,  machine  vision,  large-scale  object  identification  across 
heterogeneous  data  sources.  One  important  trend  in  Big  Data  is  Deep 
Learning,  including  unsupervised  machine  learning  techniques  (e.g.,  neural 
networks)  for  recognizing  objects  of  interest  from  Big  Data  [9],  for  instance, 
sparse  coding  (Olshausen  &  Field,  1996)  and  self-taught  learning  (Raina  et 
al.,  2007).  The  self-taught  learning  approximates  the  input  for  unlabeled 
objects  as  a  succinct,  higher-level  feature  representation  of  sparse  linear 
combination  of  the  bases.  It  uses  the  Expectation  and  Maximization  (EM) 
method  to  iteratively  learn  coefficients  and  bases.  Deep  Learning  links 
machine  vision  and  text  analysis  smartly.  For  example,  text  analysis  Latent 
Dirichlet  Analysis  (LDA)  is  a  sparse  coding  where  a  bag  of  words  is  used  as 
the  sparsely  coded  features  for  text  (Olshausen  &  Field,  1996).  Our  methods 
Lexical  Link  Analysis  (LLA),  System-Self-Awareness  (SSA),  and 
Collaborative  Learning  Agents  (CLA)  can  be  viewed  as  unsupervised  learning 
or  Deep  Learning  for  pattern  recognition,  anomaly  detection,  and  data  fusion. 

A  recursive  data  fusion  methodology  leveraging  LLA,  SSA,  and  CLA  can  be 
employed  as  follows: 

•  An  agent  j  represents  a  sensor,  operates  on  its  own  like  a  decentralized  data 
fusion;  however,  it  does  not  communicate  with  all  other  sensors  but  only  with 
the  ones  that  are  its  peers.  A  peer  list  can  be  specified  by  the  agent. 

•  An  agent  j  includes  a  learning  engine  CLA  that  collects,  analyzes  from  its 
domain  specific  data  knowledge  base  b(t,j) — for  example,  b(t,j)  may  represent 
the  statistics  for  bi-gram  feature  pairs  (word  pairs)  computed  from  LLA. 

•  An  agent  j  also  includes  a  fusion  engine  SSA  with  two  algorithms  SSA1  and 
SSA2  that  can  be  customized  externally.  SSA1  integrates  the  local 
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knowledge  base  b(t,j)  to  the  total  knowledge  base  B(t,j)  that  can  be  passed 
along  to  its  peers  and  used  globally  in  the  recursion  in  Figure  6.  SSA2 
assesses  the  total  value  of  the  agent  j  by  separating  the  total  knowledge  base 
into  the  categories  of  patterns,  emerging  and  anomalous  themes  based  on 
the  total  knowledge  base  B(t,j)  and  generates  a  total  value  V(t,j)  as  follows: 

o  Step  1:  B(t,j)  =  SSA1(B(t-1,  p(j)),  b(t,j)); 

o  Step  2:  V(t,j)  =  SSA2(B(t,j)) 

where  p(j)  represents  the  peer  list  of  agent  j. 

•  The  total  value  V(t,j)  is  used  in  the  global  sorting  and  ranking  of  relevant 
information.  In  this  recursive  data  fusion,  the  knowledge  bases  and  total 
values  are  completely  data-driven  and  automatically  discovered  from  the 
data.  Each  agent  has  the  exact  same  code  of  LLA,  SSA,  and  CLA,  yet  has  its 
own  data  apart  from  other  agents.  This  agent  work  has  the  advantages  of 
both  decentralized  and  distributed  data  fusion.  It  performs  learning  and  fusion 
simultaneously  and  in  parallel.  Meanwhile,  it  categorizes  the  patterns  and 
anomalous  information. 

•  Spark  (2015):  Map/Reduce  is  an  analytic  programming  paradigm  for  Big 
Data.  It  consists  of  two  tasks:  (1)  the  “Map”  task,  where  an  input  dataset  is 
converted  into  key/value  pairs;  and  (2)  the  “Reduce”  task,  where  outputs  of 
the  “Map”  task  are  combined  to  a  reduced  key-value  pairs.  Apache  Spark 
could  replace  Map/Reduce  for  its  speed  and  in-memory  computation. 

•  Bayesian  Networks  with  R  and  Hadoop  (Mendelevitch,  2015):  It  is  a  data- 
driven  learning  of  conditional  probability  or  structure  learning.  It  is  a 
supervised  learning  method  but  best  for  Big  Data  with  low  dimensions.  It  is  an 
approximate  inference  good  for  Big  Data  and  Hadoop  implementation. 

We  have  also  met  the  acquisition  professionals  and  discussed  how  BDAA  can  be 
applied  to  the  DoD  acquisition  process;  the  following  is  a  summary  of  the  findings: 

1 .  In  the  current  acquisition  process,  a  small  delay  or  anomaly  in  a  contract 
negotiation  process  can  have  a  huge  impact  in  its  performance  and  can 
therefore  cost  the  government  a  lot  of  money  downstream. 

2.  It  will  be  very  useful  to  apply  BDAA  such  as  LLA  for  pattern  recognition  and 
anomaly  detection  for  these  kind  of  problems  and  make  early  warnings  and 
predictions  to  prevent  the  downstream  risks. 

3.  The  Big  Acquisition  Data  might  include  programs’  cost/EUM,  SAR,  DIMIR, 
tech  data,  people  data  from  DMBC,  even  outside  economic  environment  data 
if  the  access  is  possible. 

4.  The  causes  of  the  deviations  from  the  normal  behaviors  for  the 
programs/contracts  might  be  modeled  using  physics  (e.g.,  fluid  dynamics 
theories). 

5.  LLA’s  network  perspectives,  social  plays  among  the  nodes  and  the  System 
Self-Awareness  (SSA)  theory  may  be  used  to  lay  out  the  academic  vigor  for 
the  business  processes,  for  example,  answering  the  following  questions: 

•  Are  some  nodes  drawn  towards  some  other  nodes  because  the  other  nodes 
are  more  powerful? 

•  Is  the  preferential  attachment  growth  pattern  or  expertise  growth  pattern  can 
be  used  here? 
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How  are  the  forces  of  the  nodes  modeled  and  mapped  into  the  social  network 
settings  and  actual  business  processes? 


Conclusion 

In  this  paper,  we  show  improved  LLA  analysis  reports  and  visualizations  generated 
automatically  using  multiple  categories  of  data  sources.  These  reports  and  visualizations 
reveal  that  there  are  data  correlations  and  gaps.  LLA  is  able  to  discover  in  detail  where  the 
gaps  and  inconsistencies  of  the  data  across  multiple  data  sources  reside,  which,  in  turn,  can 
lead  to  the  identification  of  future  specific  and  productive  directions  for  further  examination 
regarding  why  gaps  occur  and  where  they  exist.  It  is  a  data-driven  method  for  pattern 
recognition,  anomaly  detection,  and  data  fusion.  It  shares  indexes,  not  data,  feasible  for 
parallel  and  distributed  processing,  adaptive  to  Big  Data  Architecture  and  Analytics  and 
needs  for  Big  Acquisition  Data. 
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